Challenges of COVID-19 Case Forecasting in the US, 2020-2021 (preprint)

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naive baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making. Author SummaryAs SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.

Forecast Intervals for Infectious Disease Models (preprint)

Forecast intervals for infectious disease transmission and mortality have long been overconfident -- i.e., the advertised coverage probabilities of those intervals fell short of their subsequent performances. Further, there was no apparent relation between how good models claimed to be (as measured by their purported forecast uncertainties) and how good the models really were (as measured by their actual forecast errors). The main cause of this problem lies in the misapplication of textbook methods for uncertainty quantification. A solution lies in the creative use of predictive tail probabilities to obtain valid interval coverages. This approach is compatible with all probabilistic predictive models whose forecast error behavior does not change "too quickly" over time.

COFFEE: COVID-19 Forecasts using Fast Evaluations and Estimation (preprint)

This document details the methodology of the Los Alamos National Laboratory COVID-19 forecasting model, COFFEE (COVID-19 Forecasts using Fast Evaluations and Estimation).

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US (preprint)

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. In 2020, the COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized hundreds of thousands of specific predictions from more than 50 different academic, industry, and independent research groups. This manuscript systematically evaluates 23 models that regularly submitted forecasts of reported weekly incident COVID-19 mortality counts in the US at the state and national level. One of these models was a multi-model ensemble that combined all available forecasts each week. The performance of individual models showed high variability across time, geospatial units, and forecast horizons. Half of the models evaluated showed better accuracy than a naive baseline model. In combining the forecasts from all teams, the ensemble showed the best overall probabilistic accuracy of any model. Forecast accuracy degraded as models made predictions farther into the future, with probabilistic accuracy at a 20-week horizon more than 5 times worse than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. f

Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S. (preprint)

Background The COVID-19 pandemic has driven demand for forecasts to guide policy and planning. Previous research has suggested that combining forecasts from multiple models into a single "ensemble" forecast can increase the robustness of forecasts. Here we evaluate the real-time application of an open, collaborative ensemble to forecast deaths attributable to COVID-19 in the U.S. Methods Beginning on April 13, 2020, we collected and combined one- to four-week ahead forecasts of cumulative deaths for U.S. jurisdictions in standardized, probabilistic formats to generate real-time, publicly available ensemble forecasts. We evaluated the point prediction accuracy and calibration of these forecasts compared to reported deaths. Results Analysis of 2,512 ensemble forecasts made April 27 to July 20 with outcomes observed in the weeks ending May 23 through July 25, 2020 revealed precise short-term forecasts, with accuracy deteriorating at longer prediction horizons of up to four weeks. At all prediction horizons, the prediction intervals were well calibrated with 92-96% of observations falling within the rounded 95% prediction intervals. Conclusions This analysis demonstrates that real-time, publicly available ensemble forecasts issued in April-July 2020 provided robust short-term predictions of reported COVID-19 deaths in the United States. With the ongoing need for forecasts of impacts and resource needs for the COVID-19 response, the results underscore the importance of combining multiple probabilistic models and assessing forecast skill at different prediction horizons. Careful development, assessment, and communication of ensemble forecasts can provide reliable insight to public health decision makers.

Recombinant SARS-CoV-2 RBD molecule with a T helper epitope as a built in adjuvant induces strong neutralization antibody response (preprint)

Without approved vaccines and specific treatment, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is spreading around the world with above 20 million COVID-19 cases and approximately 700 thousand deaths until now. An efficacious and affordable vaccine is urgently needed. The Val308 - Gly548 of Spike protein of SARS-CoV-2 linked with Gln830 - Glu843 of Tetanus toxoid (TT peptide) (designated as S1-4) and without TT peptide (designated as S1-5), and prokaryotic expression, chromatography purification and the rational renaturation of the protein were performed. The antigenicity and immunogenicity of S1-4 protein was evaluated by Western Blotting (WB) in vitro and immune responses in mice, respectively. The protective efficiency of it was measured by virus neutralization test in Vero E6 cells with SARS-CoV-2. S1-4 protein was prepared to high homogeneity and purity by prokaryotic expression and chromatography purification. Adjuvanted with Alum, S1-4 protein stimulated a strong antibody response in immunized mice and caused a major Th2-type cellular immunity compared with S1-5 protein. Furthermore, the immunized sera could protect the Vero E6 cells from SARS-CoV-2 infection with neutralization antibody GMT 256. The candidate subunit vaccine molecule could stimulate strong humoral and Th1 and Th2-type cellular immune response in mice, giving us solid evidence that S1-4 protein could be a promising subunit vaccine candidate.

Functional and druggability analysis of the SARS-CoV-2 proteome (preprint)

The infectious coronavirus disease (COVID-19) pandemic, caused by the coronavirus SARS-CoV-2, appeared in December 2019 in Wuhan, China, and has spread worldwide. As of today, more than 22 million people have been infected, with almost 800,000 fatalities. With the purpose of contributing to the development of effective therapeutics, this work provides an overview of the viral machinery and functional role of each SARS-CoV-2 protein, and a thorough analysis of the structure and druggability assessment of the viral proteome. All structural, non-structural, and accessory proteins of SARS-CoV-2 have been studied, and whenever experimental structural data of SARS-CoV-2 proteins were not available, homology models were built based on solved SARS-CoV structures. Several potential allosteric or protein-protein interaction druggable sites on different viral targets were identified, knowledge that could be used to expand current drug discovery endeavors beyond the cysteine proteases and the polymerase complex. It is our hope that this study will support the efforts of the scientific community both in understanding the molecular determinants of this disease and in widening the repertoire of viral targets in the quest for repurposed or novel drugs against COVID-19.